Method and system for caching address translations from multiple address spaces in virtual machines

ABSTRACT

A method of virtualizing memory through shadow page tables that cache translations from multiple guest address spaces in a virtual machine includes a software version of a hardware tagged translation look-aside buffer. Edits to guest page tables are detected by intercepting the creation of guest-writable mappings to guest page tables with translations cached in shadow page tables. The affected cached translations are marked as stale and purged upon an address space switch or an indiscriminate flush of translations by the guest. Thereby, non-stale translations remain cached but stale translations are discarded. The method includes tracking the guest-writable mappings to guest page tables, deferring discovery of such mappings to a guest page table for the first time until a purge of all cached translations when the number of untracked guest page tables exceeds a threshold, and sharing shadow page tables between shadow address spaces and between virtual processors.

FIELD OF THE INVENTION

The invention relates to the field of computer programming. Morespecifically, the invention relates to virtualization of guest memory ina virtual machine environment through an efficient cache of addresstranslations from multiple guest address spaces.

BACKGROUND OF THE INVENTION

Virtual machine technology allows multiple virtual machines, each havingtheir own operating system, to run on a single physical machine. Thevirtual machine is called a guest with respect to the host computer. Thehost computer has virtualization software that emulates the processorarchitecture and the hardware resources required by the software runninginside a virtual machine. The virtualization software emulates theinstructions executed by the guest and virtualizes the hardwareresources of the host machine for use by the guest.

With virtual machine technology, the guest computer system exists on thehost computer system as a pure software representation of the operationof the hardware architecture of the virtual machine. The virtualizationsoftware executing on the hardware architecture of the host computermimics the operation of the entire guest computer system. Thevirtualization software acts as the interface between the hardwarearchitecture and resources of the physical machine and the instructionsexecuted by the software (e.g., operating systems, applications, etc.)running in the virtual machine environment. In one embodiment of virtualmachine technology, the virtualized environment is created by a VirtualMachine Monitor (VMM), which is a software layer that runs directly ontop of the host hardware but below the guest software and may runside-by-side and work in conjunction with a host operating system. TheVMM can virtualize the resources of the host machine by exposinginterfaces that match the guest's hardware architecture and byintercepting and virtualizing operations performed by the guest. Thisvirtualization enables the virtualization software (and the hostcomputer system) to go unnoticed by the guest operating system runningon top of it, although this virtualization does incur a performancepenalty as it consumes host resources (e.g., the processor). Inaddition, it is desirable to keep one virtual machine separated fromother virtual machines, as well as from the host. Separation of onevirtual machine from another is useful to isolate faults such that afault in one virtual machine does not affect another virtual machine.

Computer operating systems typically provide isolation between differentapplications so that when one application fails, others are notaffected. One technique is to execute each application in a separateprocess with its own address space. With this mechanism, each process ispresented with virtual memory addresses that it can read from or writeto, and the operating system takes care of backing the pages of virtualmemory used by the application with pages of physical memory andmaintaining a map of virtual addresses (VAs) to physical addresses(PAs). Since a process can only access memory via virtual addresses, theoperating system is able to restrict the pages of physical memory that aprocess may access. When a program accesses a virtual memory address,the processor translates the virtual address into a physical address byconsulting the VA-to-PA map specified by the operating system.

One processor architecture that may be used in the host computer systemis the x86 family of processors. Modern x86 processors have a built-inmemory address map lookup mechanism that efficiently converts a virtualaddress to a physical address via a procedure known as a page tablewalk. Page tables specify a VA-to-PA map, which the operating systemmaintains for each virtual address space. FIG. 1 shows a typicalprior-art page table walk process, implemented in hardware in an x86processor.

An x86 processor with paging enabled relies on a set of page directorytables (PDTs) that point to page tables (PTs) to map virtual addressesto physical addresses. Most modern x86 processors also support PhysicalAddress Extension (PAE), which increases the amount of physical memorythe machine can support. With PAE enabled, a third-level addresstranslation lookup table called the page directory pointer table (PDPT)is also used. Finally, x86 processors with 64-bit extensions require afourth-level address translation lookup table called a page map level 4table (PML4T). In all of these variations, the approach is the same:each upper-level page table references one or more page tables for thenext level. Each tree of page tables specifies a sparse map of thevirtual address space.

FIG. 1 represents a standard x86 scheme 100 for resolving a virtualaddress 190 to a physical address. A processor control register 110specifies the physical address 115 of the page directory pointer table(PDPT). In the example of an x86 processor, the control register isknown as CR3. The top bits 192 of the virtual address provide an index125 into the PDPT to select from a multiplicity of page directory tables(PDTs) 130, 140. In the FIG. 1 example, the VA selects the PDT 140,which points to a multiplicity of page tables 160, 170. The middle bits194 in the VA indexes 145 into the PDT to select a page table 170. Aspecific entry 180 in the selected page table 170 is selected by anotherfield of bits 196 in the VA. The entry 180 is the physical addresscorresponding to the virtual address 190 being resolved, with the bottombits 198 specifying the offset into the page. In summary, the x86processor resolves a virtual address by walking down the page table treestarting at the top-level page table specified in the control register(CR3) and indexing into each page table using parts of the VA. Theselected entry in the leaf page table stores the related physicaladdress.

Modem x86 processors also feature a translation look-aside buffer (TLB)in which it caches the most recently traversed page table mappings, bystoring pairs of virtual addresses and physical addresses. Upon a memoryaccess, the processor checks if the desired VA-to-PA translation isalready cached in the TLB. If it is, the TLB hit allows the processor toskip the page table walk of FIG. 1, resulting in higher performancebecause of the lower address translation latency. If the translation isnot cached, the TLB miss requires a higher expense walk of the pagetables. The x86 architecture dictates the semantics for keeping the TLB,which is a non-coherent cache of the page tables, synchronized with thepage tables, when the operating system makes modifications to them. Apage table modification is only effective after the operating systeminvalidates the affected VAs using the INVLPG instruction or flushes theTLB of all stale VA-to-PA translations by modifying the control register(CR3). In the x86 architecture, writing to CR3 not only changes the baseof the page table tree used for address translations but also flushesthe TLB of stale translations to achieve synchronization to the pagetables. Many processors indiscriminately flush the entire TLB on a writeto CR3. However, processors with a tagged TLB associate each translationwith a particular address space (a tag) and snoop the bus for memorywrites to detect translations that become stale due to page tablemodifications, so they can retain translations across address spaceswitches by removing only the stale translations.

To isolate virtual machines, the virtualization software allocatesseparate portions of host memory to different virtual machines. However,this requires the physical addresses of the guest to be virtualized, sothat while a guest may think a page of memory is at a certain physicaladdress, that page actually resides at a different physical address inthe host memory system. An address into what the guest thinks isphysical memory is called a guest physical address (GPA), and an addressinto the host's physical memory is called a system physical address(SPA). Typically, GPAs do not correspond to SPAs. Therefore, thevirtualization software must enforce an additional translation from GPAto SPA, while preserving the translation from VA to GPA enforced by theguest operating system. Consequently, the guest page tables cannot bedirectly traversed by the x86 processor's address translation hardware,because they map VAs to GPAs, not to SPAs. On every memory access, anunoptimized VMM must manually walk the guest page tables to translatethe VA to a GPA (losing the benefit of the page table walking hardware),and then translate the GPA into an SPA using its internal tables.

To improve the efficiency of memory virtualization, an optimized VMM maytake the result of the VA-to-SPA translation it performs and cache it ina format that can be efficiently accessed in the future. One solution isfor the VMM to maintain shadow page tables (SPTs) that map guest VAs toSPAs and are walked by the processor when resolving VAs. That way, whena VA cached in the SPTs is accessed, the page table walk hardware canuse the SPTs to directly translate the VA into an SPA, thus allowing theguest to access memory without any intervention by the VMM. When the VAis not cached, the processor's address translation hardware generates apage fault, which the VMM receives and must service by walking the guestpage tables and creating a VA-to-SPA translation in the SPTs. This VMMintervention is very expensive in terms of machine cycles. Although thehardware page table walk shown in FIG. 1 may take ten to a few hundredmachine cycles, the VMM intervention may take several thousands ofmachine cycles, which is up to two orders of magnitude more costly.

VA-to-SPA translations cached in the SPTs are almost analogous toVA-to-PA translations cached in a TLB of a physical processor, becausethose VAs can be resolved without the VMM looking at the page tables.Therefore, the SPTs effectively form a virtual TLB with respect to theguest. This virtual TLB preferably has the same behavior as a physicalTLB, so it preferably flushes all stale translations whenever the guestmodifies the control register 110 in FIG. 1, which occurs whenever theoperating system switches between two processes (hence, between twodifferent address spaces with their own page table tree). On manyprocessors, the hardware TLB flushes all translations indiscriminately,and the analogous operation for the virtual TLB would be to flush allentries in the SPTs. However, the cost of repopulating a VA-to-SPA entryin the virtual TLB is up to two orders of magnitude greater than thecost of repopulating a VA-to-PA entry in a physical TLB, and the virtualTLB is much larger than a physical TLB, so the impact ofindiscriminately flushing all entries in the virtual TLB, instead offlushing only the stale entries, is significantly higher. Most operatingsystems frequently switch between address spaces and consequently flushthe TLB frequently. The impact of flushing the TLB on a physical machineis small, but in a virtual machine environment, a large percentage ofhost processor cycles may be spent re-populating the virtual TLB as aresult of TLB flushes by the guest operating system.

Thus, there is a need for a method and system to implement a highlyefficient translation from a virtual address to a system physicaladdress in a virtual machine environment. It would be advantageous toimplement a virtual TLB that retained as many cached addresstranslations as possible across both address space switches andindiscriminate flushes of the entire TLB by the guest, in a manner thatpreserves the TLB semantics of existing processor architectures.Accordingly, the virtualization software may efficiently supportexisting operating systems for those architectures without requiringchanges to those operating systems. In addition, such a virtual TLB ispreferably not be too expensive in terms of computer resources such asmemory size or processor cycles, and for this technique to be worth theadded complexity. Accordingly, the virtual TLB is preferably moreefficient than a simple virtual TLB that flushes all translations on anaddress space switch. The present invention addresses the aforementionedneeds and solves them with additional advantages as expressed herein.

SUMMARY OF THE INVENTION

Aspects of the invention solve the problems associated withsimultaneously shadowing multiple address spaces to retain mappingsacross address space switches while preserving the TLB semantics ofprocessor architectures such as x86. Embodiments of the inventionimplement methods and a system to efficiently cache translations frommultiple guest address spaces in corresponding shadow address spaces andremove only the cached translations that are stale upon an address spaceswitch or an indiscriminate flush of the TLB by the guest. The challengein removing stale translations is how to detect when the guest pagetables are modified. Once a guest page table has been modified, thetranslations in the shadow page tables are stale since they no longerreflect the corresponding VA-to-GPA map that they cache, and all staletranslations are removed on the next write to CR3 to maintain the TLBsemantics of architectures such as x86 and, in turn, allow the existingoperating systems to run unmodified in the virtual machine environment.Once the guest page table modification is detected by the VMM, it nolonger needs to be notified about subsequent modifications of that guestpage table until at least after the next write to CR3.

One aspect of the invention requires efficiently implementing a softwareversion of a tagged TLB to cache translations from multiple addressspaces in a virtual machine. An efficient virtual TLB may be provided byselectively removing only the stale VA-to-SPA translations, instead offlushing the entire virtual TLB, upon an address space switch or anindiscriminate flush of the TLB. Stale translations can be detected byintercepting the creation of guest-writable mappings to guest pagetables with translations cached in shadow page tables. The virtual TLBis able to intercept the creation of guest-writable mappings, becausethe guest cannot access a page without the VMM creating a VA-to-SPAtranslation in the SPTs for that page. A guest-writable mapping to aguest page at a particular GPA is a VA-to-SPA translation with awritable attribute in which the GPA corresponds to that SPA. Uponintercepting at least one write to a guest page, the VMM creates aguest-writable mapping to disable subsequent write intercepts on thatguest page, to reduce the cost of making multiple writes to that guestpage. Since guest-writable mappings enable changes to the guest pagethat is mapped without any intervention by the VMM, when a guest pagetable becomes guest-writable, the corresponding SPT entries are assumedto be stale and are purged upon a switch to a new address space. Thus,the software version of a tagged TLB can efficiently detect whentranslations become stale, and remove those transactions but retainnon-stale translations upon an address space switch or an indiscriminateflush of the TLB by the guest.

In another aspect of the invention, the VMM efficiently write-protectsguest page tables by tracking guest-writable mappings to every guestpage table and removing those mappings. However, the CPU and memory costof tracking guest-writable mappings to all guest pages is prohibitive,so the VMM tracks only the guest pages that have been or are likely tobe used as a guest page table and this set of tracked shadowed guestpages grows incrementally and is built dynamically based on the guestpage tables walked in resolving address translations. The cost ofwrite-protecting untracked shadowed guest page tables is alsoprohibitive, so instead, the VMM assumes those tables are a prioriguest-writable, marks translations cached from those tables as stale,and defers write-protecting them until the number of times thatuntracked guest page tables are shadowed exceed a threshold, at whichpoint all cached translations in the virtual TLB are purged towrite-protect all guest pages (as they are encountered) and the trackingof guest-writable mappings to those tables begins. When deferring thewrite-protection of untracked guest page tables, the VMM adds thoseguest pages to a list of guest pages that are candidates for beingtracked. Purging all cached translations promotes the guest pages on thecandidate list to tracked guest pages.

In another aspect of the invention, shadow page tables may be sharedbetween shadow address spaces if the corresponding guest page tables areused by multiple guest address spaces. In such an instance, each shadowpage table may be tagged with the corresponding guest page directorytable attributes and processor control register flags to preserve theaccess permissions. This technique optimizes the shadowing of guest pagetables shared between address spaces, which is common in modernoperating systems. In another embodiment, shadow page tables may beshared between virtual processors, thereby allowing each processor touse translations cached by other processors, reducing the memoryoverhead of SPTs, and having only one set of SPTs to update whenremoving stale translations and handling invalidations of specificvirtual addresses. In another embodiment, the SPTs may be placed on apurge list when they become unreferenced when a shadow address space isreclaimed, so when the corresponding GPTs are shadowed again, the VMMcan link those SPTs back into a shadow address space, bringing back inthe translations cached on those SPTs. In another embodiment, the VMMrandomly prunes the hash table, usually of fixed size, used to trackinformation on a sparse subset of guest pages, such as anyguest-writable mappings to them, in order to make pre-existing space fornew entries and prune older data, with the hash table reaching arelatively stable state at some point. This opportunistic maintenance ofthe hash table allows time-critical paths to avoid the cost of having toevict existing entries.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofexemplary embodiments, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating embodimentsof the invention, there is shown in the drawings exemplary constructionsof the invention; however, the invention is not limited to the specificmethods and instrumentalities disclosed. In the drawings:

FIG. 1 is a block diagram of a prior-art scheme of determining aphysical address from a virtual address by walking a tree of page tablesspecified by the operating system;

FIG. 2 is an example of guest page tables and shadow page tables in avirtual machine environment according to the invention;

FIG. 3 is an exemplary method to efficiently maintain multiple shadowaddress spaces according to aspects of the invention;

FIG. 4 is a collection of additional exemplary method options toefficiently use shadow page tables according to aspects of theinvention; and

FIG. 5 is a block diagram showing an exemplary computing environment inwhich aspects of the invention may be implemented.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Overview

In one virtual machine implementation, the Virtual Machine Monitor (VMM)implements a set of shadow page tables (SPTs) that caches addresstranslations in the guest page tables maintained by the guest operatingsystem. The guest page tables (GPTs) map virtual addresses (VAs) toguest physical addresses (GPAs), but the VMM makes a second-leveltranslation from GPAs to system physical addresses (SPAs). The SPTscache processor traversable VA-to-SPA translations based on the twodifferent sets of mappings, VA-to-GPA specified by the guest andGPA-to-SPA enforced by the VMM. The SPTs initially do not cache anytranslations but they build up their cache of translations on demand asthe guest accesses memory. When the guest accesses a VA for the firsttime, the processor generates a page fault and traps into the VMM,because the SPTs have no valid entries for that VA. The VMM page faulthandler walks the GPTs related to that VA and reads the VA-to-GPAtranslation. If the GPT entry is invalid, the VMM reflects a page faultexception to the guest. Otherwise, the VMM translates the GPA to an SPAand updates the SPTs to reflect that VA-to-SPA translation. When theVA-to-GPA map changes as a result of a modification to the guest pagetables, the corresponding VA-to-SPA translation cached in the SPTsbecomes stale. The VMM retains control of the page tables used by thephysical processor by setting the control register that specifies thebase of the page table tree to one of its SPTs. When the guest attemptsto set that control register to one of its GPTs, the VMM intercepts andvirtualizes this operation.

The shadow page tables effectively form a virtual translation look-asidebuffer (TLB), since memory accesses to virtual addresses cached in theSPTs can bypass the VMM's software walk of the guest page tables.However, the software process to introduce a new VA-to-SPA addresstranslation into the virtual TLB is up to two orders of magnitude moreexpensive than a hardware TLB miss. Consequently, virtual TLB missesaccount for a substantial portion of the performance overhead of a VMMthat does not utilize the method described herein for cachingtranslations from multiple address spaces. Minimizing the cost of memoryvirtualization using shadow page tables requires reducing the virtualTLB miss frequency.

To reduce the cost of memory virtualization, the virtual TLB ideallycaches more translations than a physical TLB and it retains thosetranslations as long as possible. In one aspect of the invention, asolution to achieve efficient memory virtualization and substantiallyimprove performance in a virtual machine environment is to implement avirtual TLB that retains translations from multiple address spacesacross both address space switches and indiscriminate flushes of the TLBby the guest, thus minimizing the number of cached translationsdiscarded unnecessarily. This may be accomplished with the careful useof shadow page tables and the introduction of a multiplicity of shadowaddress spaces in the VMM.

The hardware translation look-aside buffer (TLB) in a physical processoris analogous to the shadow page tables in the VMM, because it cachestranslations from VAs to PAs to avoid an address translation processhaving to walk the page tables on every memory access. When the accessedVA is not cached in the TLB, a TLB miss occurs and the processor walksthe page tables starting from the base of the page table tree specifiedby the operating system (or the VMM). Since the TLB is defined by theprocessor architecture as a non-coherent cache of the page tables, theoperating system is responsible for notifying the processor of changesto the VA-to-GPA map to synchronize stale translations to the pagetables; one implementation is to remove the stale translations from theTLB. Many processor architectures provide hardware instructions toinvalidate cached translations at a few granularities, such asinvalidating a single translation and invalidating all translations.Many x86 processors invalidate all (non-global) cached entries when theoperating system switches between processes by modifying the controlregister (CR3) that specifies a pointer to the base of the page tabletree. In the x86 processor architecture, a write to CR3 is accomplishedusing the MOV CR3 instruction.

Likewise, the VMM virtualizes the processor architecture's instructionsfor invalidating page table entries and update the shadow page tables(SPTs) accordingly to remove the stale translations upon an addressspace switch, since the shadow page tables effectively form a softwarevirtual TLB and are thus a non-coherent cache of the guest page tables.However, the VMM should avoid needlessly invalidating all of thetranslations cached in the SPTs, because otherwise, it incurs thesignificant cost of rebuilding that cache by taking VMM page faults onmemory accesses by the guest. This means that an efficient virtual TLBcannot use the simple approach of discarding all cached translationswhenever the guest switches address spaces.

One approach for multiple address space shadowing at a high level may beintroduced by describing what an equivalent hardware solution would dobefore a similar but more complex solution for software is described.Some x86 processors implement a tagged TLB that caches translations frommultiple address spaces. In such processors, the TLB is enhanced withtag bits. These bits are associated with every TLB entry, and only theentries whose tag matches the current tag register would be used toresolve VA-to-PA mappings. It is possible to use the base of the pagetable tree (CR3) value as the tag, but this would be an expensivehardware undertaking. The bits used to represent a tag can be reduced(e.g., three bits can represent up to eight address spaces) if a smallcontent-addressable memory is used to track recent CR3 values. Thetagged bits enable the TLB to distinguish translations from differentaddress spaces and to lookup only the translations for the currentaddress space.

With tagged TLBs, upon a MOV CR3 instruction, the x86 processor modifiesthe current tag register set to the tag corresponding to the new CR3,and the TLB will then resolve VA-to-PA mappings with the new tag. Newentries introduced by a TLB miss resolution are tagged with the currenttag value. The net result is that the impact of MOV CR3, which typicallyinvalidates all (non-global) TLB entries on processors without taggedTLB, is reduced since any entries associated with the previous CR3 arenot flushed.

To maintain the semantics of a TLB flush, the tagged TLB remainssynchronized with the corresponding page table entries at certain statechanges. The x86 architecture provides the INVLPG instruction toexplicitly invalidate a specific VA-to-PA mapping in the TLB, but thesemantics of MOV CR3 implies flushing all (non-global) TLB entries inaddition to switching address spaces. The tagged TLB scheme invalidatesthe specified VA in response to an INVLPG instruction. It also performsinvalidations implied by MOV CR3 in response to modifications to bothactive and inactive page tables. The TLB entries are synchronized to thepage tables upon a MOV CR3, meaning that any stale translations areremoved.

To detect page table modifications, a second physical address TLB(PA-TLB) may be created to track the PAs of the page tables used to formthe VA-to-PA mappings in the TLB. Whenever a TLB miss causes asuccessful page table walk, an entry is created in the PA-TLB to trackthe PA of the page table entry itself as well as a reference to the newTLB entry. Physical addresses from all processors are then snooped onthe bus. When the physical address of a memory write matches a PA in thePA-TLB, the TLB entries corresponding to the modified VA-to-PA mappingare invalidated as they may have become stale. Any entries evicted fromthe PA-TLB also require the corresponding TLB entries to be removed,since evicting the PA-TLB entries prevents the processor from monitoringthe corresponding page table entries for changes.

Consequently, a hardware solution would maintain TLB entries associatedwith multiple address spaces. Since MOV CR3 implies a TLB flush, thehardware would have to intercept page table modifications and invalidateTLB entries that become stale as a result. However, the write snoopingon the bus allows the hardware to immediately detect and invalidatestale translations. Under this scheme, MOV CR3 no longer impliessynchronizing the TLB because stale TLB entries have already beenpurged.

A hardware implementation of tagged TLBs has the advantage of being ableto snoop on the bus for writes from all processors and perform lookupsin parallel. An analogous software implementation would write-protectall mappings to shadowed page tables, but this is much more expensive.To cache multiple address spaces, the VMM maintains multiple SPT treeseach of which is associated with a tag. Each of these SPT trees iscalled a shadow address space (SAS). Only the SPT tree whose tag matchesthe current tag is used to resolve VA-to-SPA translations. As with thehardware solution, the tag could be only a few bits and a lookup tableis used to map recent CR3 values to a tag. Upon a MOV CR3 by the guest,the physical CR3 is set to point to the SPT tree corresponding to thenew tag. Page faults and INVLPG instructions would affect entries in thecurrent SPT tree. This scheme enables the VMM to preserve VA-to-SPAmappings across CR3 modifications.

However, as with the hardware TLB, this multiple address space shadowingscheme preferably honors the TLB flush implied by MOV CR3 andsynchronizes the SPT entries with the corresponding guest page tableentries. To detect stale cached translations, the VMM intercepts allguest page table modifications. Since the guest cannot edit a guest pagetable before the VMM creates a VA-to-SPA translation to that GPT, theVMM can detect the creation of such mappings when handling page faultsand it can create a write-prohibited mapping.

The VMM maintains a hash table that is analogous to the hardwaresolution's PA-TLB for tracking the guest pages that contain shadowedguest page tables. When servicing a page fault, the VMM checks the hashtable to determine whether the guest page being accessed, andsubsequently about to be mapped, contains a guest page table. If that isthe case, the corresponding SPT is marked for flushing upon the next MOVCR3 since the GPT is now guest-writable and the SPT entries may becomestale. In essence, upon a MOV CR3, the VMM flushes the SPT of any guestpage table that is both guest-writable and shadowed to ensure that thevirtual TLB does not cache any stale translations when the MOV CR3virtualization completes.

However, a serious complication is that the guest page containing theguest page table may have been mapped before it is shadowed by the VMM.To address this problem, the VMM records for each guest page whetherthat page is mapped in the SPTs. Stated another way, the VM records foreach page whether the page is guest-writable. A guest-writable page is apage that a guest has unconstrained access to change without the VMMbeing aware of the change. Before it shadows any guest page table, theVMM checks to see whether that guest page is mapped. If that is thecase, the VMM either modifies any mappings to that guest page so theguest cannot modify the page without the VMM intercepting it, or itmarks the SPT to be flushed upon the next MOV CR3 since it may havestale entries due to existing guest-writable mappings. The VMM tracksthe guest-writable mappings to each guest page that is used as a GPT, soit can efficiently write-protect the GPT by removing those mappings.

This solution satisfies the implied TLB synchronization that occurswhenever the guest executes a MOV CR3 instruction. The key invariant isthat any guest page table that is ever simultaneously guest-writable andshadowed will have any corresponding SPTs flushed upon the next MOV CR3.The solution in software is much more complicated, because hardware canbe more fine-grained by snooping for memory writes on bus, whereassoftware can only intercept the creation of guest-writable mappings toshadowed guest page tables.

EXEMPLARY EMBODIMENTS OF THE INVENTION

FIG. 2 is an example of a set 200 of guest page tables and correspondingshadow page tables according to the present invention. A hardwareprocessor control register 202, such as an x86 family CR3 register, maybe loaded with a physical address 204 to a guest page directory table(GPDT) 210. The GPDT 210 entry for a given virtual address, for exampleVA2, provides a reference 216 to a guest page table (GPT2) whichprovides a reference 218 to the corresponding guest physical addressGPA2. The VMM obtains the address GPA2 from walking the GPTs andtranslates it into a system physical address SPA2. As an aspect of thepresent invention, a subset of the guest page table entries arereflected in the shadow page tables. Shadow page directory table 240(SPDT) is a shadow copy of GPDT 210. Shadow page table 1 (SPT1) 250 is ashadow copy of GPT1 220 and shadow page table 2 (SPT2) 260 is a shadowcopy of GPT2 230. The shadow page table entries are such that a virtualaddress VA2 in GPDT 210 corresponding to GPA2 218 in GPT2 230 isshadowed by VA2 in SPDT 240 corresponding to SPA2 248 in SPT2 260.

Coherency issues arise if an edit in the guest page tables changes themapping of a virtual address to a different GPA or modifies anyattribute in the guest page table. For example, if the mapping of VA1 inGPT1 220 is edited such that VA1 is translated to GPA2, then theVA1-to-SPA1 translation cached in the shadow page table 250 no longerreflects the VP-to-GPA translation in GPT1 220. Cached translations thatare stale such as the VA1-to-SPA 1 translation should be purged from theshadow page tables when the guest performs an operation, which in theemulated architecture, requires a flush of all stale translations. But,since the VA2-to-SPA2 translation in the SPT2 still corresponds to theVA2-to-GPA2 translation in GPT2, that SPT entry can be retained.

In one aspect of the invention, address translations cached in shadowpage tables are maintained so that they remain consistent with thecorresponding shadowed guest page tables to the extent that is necessaryin order to maintain proper execution of the virtual machine'sinstructions, as dictated by the processor architecture. Addresstranslations are cached for multiple address spaces, and are maintainedin such a way that as many cached translations as possible are retainedacross certain guest processor events that require certain staletranslations to be removed from the shadow page tables.

A “shadowed” guest page table is a GPT that has one or more of itstranslations cached in some SPT. A page of guest memory is “mapped” ifthere is at least one entry in an SPT corresponding to that page. When aguest page is mapped, the guest can access that page without the VMMintercepting the access, because the hardware page table walker is ableto locate the page of memory using the VA-to-SPA translation in theSPTs. In the x86 processor architecture, page tables are stored in pagesof physical memory, and likewise in a virtual machine, guest page tablesare stored in pages of guest physical memory. Therefore, it is possiblefor a page of guest physical memory containing a guest page table to beboth shadowed and mapped at the same time. This case gives rise to thecache coherency issues, because it allows the guest to make changes tothat guest page table without the VMM being notified, but the VMM stillneeds to maintain consistency between each SPT and the GPT it shadows.

In one aspect of the invention, the manner in which SPTs cache addresstranslations corresponding to the translations found in guest pagetables on a virtual machine is analogous to the manner in which a TLBcaches address translations found in page tables on a physical machine.As such, in order to maintain correct operation of a virtual machine,the validity of the translations in the SPTs with respect to thecontents of the guest page tables on a virtual machine are subject tothe same requirements as is the validity of the translations in the TLBwith respect to the page tables on a physical machine.

On a physical machine, in some processor architectures including x86,cached translations in a TLB are permitted to be stale with respect tothe address translations that appear in the page tables for a period oftime after the page tables are modified. Translations in the TLB becomesynchronized with the translations in the page tables again after anaddress space switch, which occurs when the operating system writes toCR3. In one embodiment, the VMM maintains correct behavior by purgingstale mappings from the SPTs when the guest operating system loads avalue into the virtual processor's CR3.

In one aspect of the invention, the VMM maintains multiple shadowaddress spaces, which include a tree of shadow page tables. Each shadowaddress space corresponds to an address space within the guest. The SPTsof each shadow address space cache translations from the guest pagetables of the corresponding guest address space.

The guest creates a mapping in its page tables in order to edit one ofits own guest page tables. The VMM can detect when a correspondingVA-to-SPA mapping to the guest page table is created in the SPTs whenhandling a page fault. Although the VMM cannot directly interceptindividual writes to guest page tables by way of physical addresssnooping, as a tagged TLB in hardware can, it can intercept the creationof a guest-writable mapping that would permit the guest to make suchmodifications without trapping into the VMM. As guests may edit manyentries in a guest page table at a time, after one or more writeintercepts, the VMM chooses to create a guest-writable mapping at thatVA to prevent further intercepts until the next time the guest pagetable is shadowed, at which point the write intercepts could bere-enabled by removing those guest-writable mappings. The VMM keepstrack of which guest pages contain guest page tables that are currentlyshadowed. Upon a page fault, the VMM checks to see if the guest pagebeing mapped in is also currently shadowed. If so, the SPT shadowing theGPT being mapped in is marked as stale and purged upon the next addressspace switch, since the GPT page is now guest-writable and the SPTentries may no longer reflect the translations in the GPT. This sequenceof events is referred to as shadow-then-map.

However, a complication is that a guest page containing a guest pagetable may have guest-writable mappings before it is shadowed by a SPT inthe VMM. This is problematic, because the guest can use the existingguest-writable mapping to modify the guest page table, causing theentries in the SPT to become stale. This sequence of events is referredto as map-then-shadow. To address this occurrence, before the VMMshadows a guest page table, the VMM prohibits undetected writes to theguest page table by removing all guest-writable mappings to that page inthe SPTs (so the guest cannot modify the page undetected by the VMM) orimmediately marks all translations in the SPT as stale. Bywrite-protecting the guest page tables, the VMM can trap and thus detectany attempted change which alerts the VMM to potential staletranslations in the SPTs. The alternative to write-protecting the GPT isto assume that the SPT has stale translations, but this means thosetranslations will be lost on the next address space switch orindiscriminate flush of the TLB.

The VMM purges the SPT corresponding to any guest page table that maycontain stale entries when switching to a shadow address space (SAS).This action ensures that the SPTs are synchronized to the guest pagetables by the time the switch to a new SAS completes. Therefore, the VMMpage fault handler detects two conditions that result in SPTs with staleentries: shadow-then-map and map-then-shadow. In shadow-then-map, theVMM detects that the page being mapped in is a shadowed GPT. Inmap-then-shadow, the VMM detects that guest-writable mappings to the GPTbeing shadowed exist in the SPTs. Upon an address space switch, the VMMmay then selectively purge the SPTs marked stale of invalid addresstranslations so that all of the remaining VA-to-SPA address translationsin the virtual TLB are accurate. This solution satisfies the implied TLBsynchronization that may be required upon an address space switch.

In another aspect of the invention, guest-writable mappings to specificpages may be efficiently removed in the course of write-protecting aguest page. In this aspect of the invention, the guest-writable mappingsare tracked to only a specific but evolving subset of guest pages. Overtime, a working set of the guest pages that are of interest (i.e., havebeen or will likely be used as a guest page table) will be built up inthe VMM.

The VMM removes all of the guest-writable mappings to a given pagebefore shadowing a guest page table. This allows the VMM to avoidmarking the SPT as stale due to the possibility that the guest might beable to modify that GPT through an existing guest mapping unknown to theVMM as shown and described in FIG. 2 as the VA1-to-SPA1 translation,which allows the guest to modify GPT2 undetected by the VMM. Thus, astale SPT is one that may no longer accurately reflect the translationsin the corresponding guest page table. However, tracking theguest-writable mappings to every guest page is infeasible due to thesubstantial amount of memory and CPU time needed to interrogate ormaintain the database of guest-writable mappings. This scheme wouldinvolve keeping multiple back references from every guest page to theguest-writable VA-to-SPA mapping to that guest page, which could be upto one million pages for 4 GB of 4 KB pages. The fact that only a smallsubset of guest pages is actually of interest to the virtual TLBmotivates a novel approach to solving this problem.

The VMM watches for guest-writable mappings only to pages that havepreviously been shadowed since such guest pages are likely to be used asa guest page table. When it first shadows a guest page table, it insertsthat page into a hash table indexed by GPA. When creating aguest-writable mapping to a page, the VMM checks whether that guest pageis in the hash table. If it is, the VMM records in the hash table a backreference from that guest page to the SPT entry mapping the guest pagein. This enables the VMM to quickly find and write-protect allguest-writable mappings to any given page tracked by the hash table.

A complication arises when a page with a GPT is shadowed for the firsttime, since the VMM has not been watching for guest-writable mappings tothat page. The VMM inserts the page into the hash table so that any newmappings will be recorded, but it does not know what guest-writablemappings to the page already exist. It is too costly to scan all SPTentries for such mappings to make this determination. Instead, itassumes that the page may have guest-writable mappings to it that areunknown to the VMM, and it also records in the hash table that the pageshould be tracked moving forward. Any shadow pages tables correspondingto that GPT will be considered stale. Periodically, the VMM purges someor all guest mappings in the SPTs so that all pages tracked in the hashtable have known guest-writable mappings, since there will be no guestmappings after the full-purge of the virtual TLB.

Using this approach, the VMM will build up a working set of GPT pages inthe hash table, at which point it will be tracking guest mappings tomost or all shadowed GPTs. Data and simulations of many workloadsindicated that the set of pages with GPTs, though evolving, isrelatively stable in steady state, with typically at most a few pagesper second becoming GPTs. This ensures that the VMM will not need toperform a full purge of all cached translations very frequently. The VMMperforms a full-purge of the virtual TLB only when the number of timesan untracked GPT is shadowed exceeds a certain threshold.

Overall, the present invention may be considered optimized based onseveral assumptions concerning the overall process. Those assumptionsinclude the idea that the vast majority of memory accesses originatefrom a small number of address spaces. Next, the set of guest pages usedas page tables is fairly stable in steady state, and the number of guestpage tables is much smaller than the number of guest pages.Additionally, the guest page tables may be frequently modified by theguest.

As part of the embodiment, the VMM maintains metadata on each shadowpage table, including which GPT page it shadows, whether it has stalemappings, and which SPDT entries point to the SPT. An SPT becomes staleif the corresponding guest page table page becomes guest-writable, andthe VMM updates the SPT metadata accordingly.

FIG. 3 depicts a flow diagram of an exemplary method 300 in accordancewith aspects of the invention. The method 300 caches translations ofvirtual addresses to system physical addresses from multiple addressspaces in a virtual machine to retain as many translations as possibleacross address space switches. The method 300 includes write-protectinga guest address space (step 310) by removing all guest-writable mappingsto the guest pages tables before caching one of the guest page tabletranslations in a shadow page table. Next, writes to guest page tablesare intercepted (step 320). After one or more write intercepts on aguest page table, a guest-writable mappings is made to the shadowedguest page table, so the translations in the corresponding shadow pagetables are marked as being stale. The marked translations are thenpurged upon an address space switch (step 330).

Optional methods to the main method of steps 310-330 are available asaspects of the invention. For example, optional method step 340(connector 330 to 340 shown dotted) is a start of one optional methodfrom step 330 as are optional steps 410, 430, 450 and 470. Theguest-writable mappings to the set of tracked guest page tables aretracked (step 340). The set of tracked guest page tables is built updynamically based on the guest page tables walked in resolving addresstranslations. The optional method step 340 enhances the efficiency ofwrite-protecting GPTs. Write-protecting tracked GPTs involves removingthe tracked guest-writable mappings to them, but write-protectinguntracked GPTs involves scanning all SPTs to find guest-writablemappings to those GPTs. In one embodiment, write-protecting untrackedGPTs is deferred, and their corresponding SPTs are marked as stale asthose GPTs may have guest-writable mappings (step 350). The untrackedGPTs that are shadowed are placed on a list of guest pages that arecandidates for being tracked. When the number of times that untrackedGPTs are shadowed exceeds a threshold, the method purges all cachedtranslations, thus write-protecting all guest pages, and begins trackingguest-writable mappings to the previously untracked GPTs on thecandidate list (step 360).

In addressing another aspect of the invention, the virtual TLB can befurther optimized on processor architectures such as x86 and x86-64 thatsupport global mappings, which are translations that are common to alladdress spaces. These translations do not need to be flushed from thetranslation look-aside buffer (TLB) when the operating system modifiesthe base of the page table tree upon an address space switch. Thisretention of mappings helps to avoid TLB misses due to global mappingsbeing invalidated in the TLB.

The VMM can share SPTs between shadow address spaces. This sharingallows virtual processors to load in global mappings cached on sharedpage tables in a batched fashion into every single shadow address spaceupon the first page fault on a virtual address translated by that sharedpage table, reducing the number of page faults that occur when comparedto an implementation that does not share SPTs between shadow addressspaces. When a fault occurs, the VMM looks to see if there is an SPTthat already caches the translation. If there is, the SPT is linked intothe SPDT of the current shadow address space, thus bringing in the othertranslations on that SPT in bulk.

As an aspect of the present invention, the VMM may detect when globalmappings in the SPTs become stale, just as it does with non-globalmappings as mentioned above. Therefore, upon a modification to the PGEbit of the CR4 control register, which requires both global andnon-global translations that are stale to be flushed, the shadow pagetables can be synchronized to the guest page tables in the same way thatthey are synchronized on an address space switch resulting from a writeto CR3. This allows the virtual TLB to avoid having to perform anindiscriminate flush of the entire TLB upon a change to the PGE bit ofCR4, which x86 operating systems use to synchronize the global mappings.Avoiding an indiscriminate purge of the SPTs preserves many mappingsthat are still valid in shadow page tables. This results in longerretention of global mappings cached in the shadow page tables.

It is noted that one complication may arise in sharing SPTs betweenshadow address spaces. The VMM may propagate attribute flags in theupper-level guest page tables such as the guest page directory table(GPDT) into the lowest-level SPT, so the SPT effectively cachesinformation not just from the GPT but also from the GPDT referencing it.Thus, each SPT shadows not only a specific GPT but also a specific setof GPDT entry attributes. However, as an aspect of the invention, theVMM takes this into account when looking for an existing SPT thatshadows a particular GPT, by also checking to see if the shadow pagetable shadows the desired GPDT entry attributes. This is preferred toensure correctness, since each shadow address space must preserve theaccess permissions specified throughout the guest page table tree evenif it shares the SPTs with other shadow address spaces. This sametechnique is also applied to the processor control register flags, suchas the PSE bit in the CR4 register which determines whether large pagesare supported by the processor. Thus, each SPT also shadows a particularset of control register flags, and those flags are taken into accountwhen looking for an existing SPT that shadows a particular GPT.

In another aspect of the invention, the shadow page tables and even theshadow address spaces can be shared between different virtualprocessors. In a manner similar to the sharing of SPTs between shadowaddress spaces on the same virtual processor described earlier, when apage fault occurs, the virtual processor first looks for an existing SPTthat may already contain the cached translation and if one exists, linksit into its current SAS, thereby bringing in all address translationscached in that SPT. Therefore, when an address translation is cached onone virtual processor, it is available to other virtual processors aswell. Similarly, the virtual processors can share the shadow addressspaces currently in the virtual TLB, so when an address space switchoccurs, the virtual processor looks for an existing SAS that shadows theguest address space being switched to. In this respect, the softwaresolution differs from the hardware solution, in which there typicallyexists one TLB for each processor and address translations cached in theTLBs are not shared between processors.

Sharing SPTs in this manner reduces cross-processor TLB shootdown costby eliminating the need for communication between virtual processors viaan inter-processor interrupt, because the virtual machine uses a singlevirtual TLB that is shared between the virtual processors. In addition,the VMM only needs to update one set of shadow page tables when it hasto synchronize the virtual TLB. If the shadow page tables and shadowaddress spaces were not shared, it is likely that the VMM will have tomake updates to multiple sets of SPTs in response to an address spaceswitch, an indiscriminate flush of the TLB by the guest, or aninvalidation of a translation for a particular virtual address.Furthermore, sharing SPTs between virtual processors leads to lowermemory usage.

In some circumstances in a virtual machine, multiple workloads supportedby multiple virtual processors may have more active processes andcorresponding address spaces than there are shadow address spaces in theVMM. A good example is a Windows terminal server on which many users runseveral processes. In the worst case, the virtual processor neverswitches to a shadowed address space before it is evicted, thuseffectively gaining no benefit from caching translations from multipleaddress spaces.

In one embodiment, a purge list of SPTs from evicted shadow addressspaces may be maintained. When shadow address spaces are evicted, theirshadow page tables can be placed on a free list from which SPTs areallocated. However, a more sophisticated approach is to place SPTs withno stale translations on a purge list such that the SPTs are notre-allocated for use in another shadow address space right away. Both afree list and a purge list may be maintained. When a virtual processoreventually switches back to that address space or caches a translationfrom the corresponding GPTs, those SPTs and their cached translationsmay be on the purge list. If they are, the VMM can link an SPT back intothe shadow page directory table of the shadow address space, thusbringing back in all of the translations cached by that SPT.Additionally, SPTs may be removed from the purge list if an SPT needs tobe allocated but the free list is empty. This novel techniqueeffectively uses the larger space of SPTs to retain translations fromaddress spaces that technically are no longer shadowed, with the hopethat the address space will become shadowed before the SPT is reused toshadow a different GPT. This enables the VMM to more gracefully supportguests using more address spaces than the VMM has shadow address spaces.

In another aspect of the invention, on-demand random pruning of adatabase for tracking information on guest pages may be accomplished. Inone example embodiment, the VMM can implement this database with a hashtable to efficiently track information on a sparse subset of guestpages. In this instance, there may be a fixed number of objects fortracking guest pages and the guest-writable mappings to them. In thissituation, the VMM may evict one of those objects at random to prune thehash table of older information. As a result, the database may reach arelatively stable state at some point. Additionally, if the databasebecome large, the VMM may randomly purge the tracked information on aguest page to avoid the expense of the maintaining the database.

FIG. 4 depicts optional embodiment methods 400 that may be exercised inaddition to the method of FIG. 3. The optional method step 410 uses theshadow page tables more efficiently by reusing previously cachedtranslations. A shadow page table may be shared between shadow addressspaces (step 410) if the corresponding guest page table is used inmultiple guest address spaces. Each shadow page table (leaf node) may betagged with guest page directory (intermediate node) attributes andprocessor control register flags to preserve access permissions (step420). In another option, shadow page tables and even shadow addressspaces may be shared between virtual processors in a virtual machine(step 430). This allows for the efficient handling of invalidations ofguest page table entries by having only one set of shadow page tableswhich are shared between different processors to update (step 440). Inanother option, shadow page tables may be placed on a purge list (step450) when they are no longer referenced by a shadow address space as aresult of a shadow address space being reclaimed. However, a subsequentguest memory access may use an address translation from a guest pagetable corresponding to a shadow page table on the purge list. If so,that shadow page table may be linked back into a shadow address space(step 460), bringing in the cached translations on that shadow pagetable in bulk. In another option, the database that tracks informationon a sparse subset of guest pages such as the guest-writable mappings tothose guest pages may be randomly pruned (step 470). This pruning makesspace for new entries and to prune older data and avoids having to evictexisting entries on time-critical paths. Eventually, the database mayreach a relatively stable state as it builds up a working set ofinformation.

In one example embodiment of the invention, a VMM page fault handler hasto detect two cases: map-then-shadow and shadow-then-map. The purpose isto mark any shadow page tables caching translations from a guest pagetable with guest-writable mappings to it as stale. The CR3 reloadhandler has to locate a SAS shadowing the guest address space beingswitched to, else evict an existing SAS. If it finds an existing SAS,the VMM must make sure the SAS it switches to does not have stalemappings. The INVLPG handler has to remove the cached translation forthe specified virtual address from the shadow page tables.

In one embodiment, upon a VMM page fault due to a guest memory access ina virtual machine emulating the x86 processor architecture, the VMMwalks the GPT tree to the fault virtual address (VA) to obtain the faultGPA. If there is no VA-to-GPA translation at that virtual address, theVMM signals a page fault to the guest. Otherwise, the fault GPA may thenbe translated into a SPA. If the SPDT entry for the fault VA is notpresent or the GPDT entry has changed, the VMM may look for an existingSPT that shadows the GPT used to resolve the page fault in the hashtable, as well as the GPDT entry attributes and the guest processorcontrol register flags. If a GPT-to-SPT map is found, the VMM mayconnect the SPDT entry to the SPT shadowing that GPT. Otherwise, the VMMmay allocate a new SPT and add a GPT-to-SPT map in the hash table. Ifthe GPT is not already tracked by the hash table, the VMM may add it andmark it as being partially tracked, meaning that there may beguest-writable mappings to it that are unknown to the VMM. If the GPT ispartially tracked, the VMM must mark the corresponding SPT as stale dueto the unknown guest-writable mappings to that GPT. If the GPT is in thehash table, the VMM may efficiently write-protect that GPT by removingall of the guest-writable mappings to that GPT page by removing thecorresponding VA-to-SPA translations in the shadow page tables. Then,the VMM may connect the SPDT entry to the SPT and record the backreference from the SPT to the SPDT entry. If the hash table is trackingthe fault GPA and the page fault was on a write by the guest, the VMMrecords in the hash table a guest-writable mapping to the GPT from theSPT entry and mark any SPTs shadowing that GPT as stale. Finally, theVMM creates the VA-to-SPA mapping in the SPT entry.

In one embodiment of the invention, upon the guest executing a MOV CR3instruction to switch address spaces in a virtual machine emulating thex86 processor architecture, if too many GPTs are partially tracked, theVMM can flush the cached translations in all of its SPTs and mark allGPT pages in the hash table as fully tracked. If the new guest CR3 isnot currently shadowed by a SAS, the VMM can choose an existing SAS toevict, remove all of its mappings, and associate the evicted SAS withthe new guest CR3. Otherwise, the VMM can lookup the SAS for the newguest CR3, invalidate the SPDT entries for the GPDT entries that havechanged, and invalidate the SPDT entries to SPTs that are stale.Finally, the VMM loads the base of the SPT tree for the SAS into thephysical CR3.

In one embodiment of the invention, upon the guest executing an INVLPGinstruction in a virtual machine emulating the x86 processorarchitecture, the VMM can invalidate the SPT entry at the VA specifiedby the INVLPG instruction and execute a physical INVLPG on the VA toremove the entry from the physical TLB.

Exemplary Computing Device

FIG. 5 and the following discussion are intended to provide a briefgeneral description of a suitable computing environment in whichembodiments of the invention may be implemented. While a general-purposecomputer is described below, this is but one single processor example,and embodiments of the invention with multiple processors may beimplemented with other computing devices, such as a client havingnetwork/bus interoperability and interaction. Thus, embodiments of theinvention may be implemented in an environment of networked hostedservices in which very little or minimal client resources areimplicated, e.g., a networked environment in which the client deviceserves merely as an interface to the network/bus, such as an objectplaced in an appliance, or other computing devices and objects as well.In essence, anywhere that data may be stored or from which data may beretrieved is a desirable, or suitable, environment for operation.

In the context of a virtual machine environment, the computingenvironment of FIG. 5 may be implemented with one or more processorswhere a host operating system or virtualization software may service amultiplicity of different virtual machines. In this context, theconfiguration of FIG. 5 provides an exemplary instance of a single CPUsystem with the capability to host multiple virtual machines, eachhaving an operating system and requesting hardware resources from thehost computer of FIG. 5.

Although not required, embodiments of the invention can also beimplemented via an operating system, for use by a developer of servicesfor a device or object, and/or included within application software.Software may be described in the general context of computer-executableinstructions, such as program modules, being executed by one or morecomputers, such as client workstations, servers or other devices.Generally, program modules include routines, programs, objects,components, data structures, and the like that perform particular tasksor implement particular abstract data types. Typically, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. Moreover, those skilled in the art willappreciate that various embodiments of the invention may be practicedwith other computer configurations. Other well-known computing systems,environments, and/or configurations that may be suitable for useinclude, but are not limited to, personal computers (PCs), automatedteller machines, server computers, hand-held or laptop devices,multi-processor systems, microprocessor-based systems, programmableconsumer electronics, network PCs, appliances, lights, environmentalcontrol elements, minicomputers, mainframe computers, and the like.Embodiments of the invention may also be practiced in distributedcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network/bus or otherdata transmission medium. In a distributed computing environment,program modules may be located in both local and remote computer storagemedia including memory storage devices and client nodes may in turnbehave as server nodes.

FIG. 5 thus illustrates an example of a suitable computing systemenvironment 500 in which the embodiments of the invention may beimplemented, although as made clear above, the computing systemenvironment 500 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of an embodiment of the invention. Neither should thecomputing environment 500 be interpreted as having any dependency orrequirement relating to any one or combination of components illustratedin the exemplary operating environment 500.

With reference to FIG. 5, an exemplary system for implementing anembodiment of the invention includes a general-purpose computing devicein the form of a computer system 510. Components of computer system 510may include, but are not limited to, a processing unit 520, a systemmemory 530, and a system bus 521 that couples various system componentsincluding the system memory to the processing unit 520. The system bus521 may be any of several types of bus structures including a memory busor memory controller, a peripheral bus, and a local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus (also known as Mezzanine bus).

Computer system 510 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computer system 510 and includes both volatile andnonvolatile, removable and non-removable media. By way of example, andnot limitation, computer-readable media may comprise computer storagemedia and communication media. Computer storage media includes volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data. Computer storage media includes, but is not limited to,Random Access Memory (RAM), Read Only Memory (ROM), ElectricallyErasable Programmable Read Only Memory (EEPROM), flash memory or othermemory technology, Compact Disk Read Only Memory (CD-ROM), compactdisc-rewritable (CD-RW), digital versatile disks (DVD) or other opticaldisk storage, magnetic cassettes, magnetic tape, magnetic disk storageor other magnetic storage devices, or any other medium which can be usedto store the desired information and which can accessed by computersystem 510. Communication media typically embodies computer-readableinstructions, data structures, program modules or other data in amodulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. Combinations of any of the above should also be includedwithin the scope of computer-readable media.

The system memory 530 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 531and random access memory (RAM) 532. A basic input/output system 533(BIOS), containing the basic routines that help to transfer informationbetween elements within computer system 510, such as during startup, istypically stored in ROM 531. RAM 532 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 520. By way of example, and notlimitation, FIG. 5 illustrates operating system 534, applicationprograms 535, other program modules 536, and program data 537.

The computer system 510 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 5 illustrates a hard disk drive 541 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 551that reads from or writes to a removable, nonvolatile magnetic disk 552,and an optical disk drive 555 that reads from or writes to a removable,nonvolatile optical disk 556, such as a CD-ROM, CD-RW, DVD, or otheroptical media. Other removable/non-removable, volatile/nonvolatilecomputer storage media that can be used in the exemplary operatingenvironment include, but are not limited to, magnetic tape cassettes,flash memory cards, digital versatile disks, digital video tape, solidstate RAM, solid state ROM, and the like. The hard disk drive 541 istypically connected to the system bus 521 through a non-removable memoryinterface such as interface 540, and magnetic disk drive 551 and opticaldisk drive 555 are typically connected to the system bus 521 by aremovable memory interface, such as interface 550.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 5 provide storage of computer-readableinstructions, data structures, program modules, and other data for thecomputer system 510. In FIG. 5, for example, hard disk drive 541 isillustrated as storing operating system 544, application programs 545,other program modules 546, and program data 547. Note that thesecomponents can either be the same as or different from operating system534, application programs 535, other program modules 536, and programdata 537. Operating system 544, application programs 545, other programmodules 546, and program data 547 are given different numbers here toillustrate that, at a minimum, they are different copies. A user mayenter commands and information into the computer system 510 throughinput devices such as a keyboard 562 and pointing device 561, commonlyreferred to as a mouse, trackball, or touch pad. Other input devices(not shown) may include a microphone, joystick, game pad, satellitedish, scanner, or the like. These and other input devices are oftenconnected to the processing unit 520 through a user input interface 560that is coupled to the system bus 521, but may be connected by otherinterface and bus structures, such as a parallel port, game port, or auniversal serial bus (USB). A monitor 591 or other type of displaydevice is also connected to the system bus 521 via an interface, such asa video interface 590, which may in turn communicate with video memory(not shown). In addition to monitor 591, computer systems may alsoinclude other peripheral output devices such as speakers 597 and printer596, which may be connected through an output peripheral interface 595.

The computer system 510 may operate in a networked or distributedenvironment using logical connections to one or more remote computers,such as a remote computer 580. The remote computer 580 may be a personalcomputer, a server, a router, a network PC, a peer device, or othercommon network node, and typically includes many or all of the elementsdescribed above relative to the computer system 510, although only amemory storage device 581 has been illustrated in FIG. 5. The logicalconnections depicted in FIG. 5 include a local area network (LAN) 571and a wide area network (WAN) 573, but may also include othernetworks/buses. Such networking environments are commonplace in homes,offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer system 510 isconnected to the LAN 571 through a network interface or adapter 570.When used in a WAN networking environment, the computer system 510typically includes a modem 572 or other means for establishingcommunications over the WAN 573, such as the Internet. The modem 572,which may be internal or external, may be connected to the system bus521 via the user input interface 560, or other appropriate mechanism. Ina networked environment, program modules depicted relative to thecomputer system 510, or portions thereof, may be stored in the remotememory storage device. By way of example, and not limitation, FIG. 5illustrates remote application programs 585 as residing on memory device581. It will be appreciated that the network connections shown areexemplary and other means of establishing a communications link betweenthe computers may be used.

Various distributed computing frameworks have been and are beingdeveloped in light of the convergence of personal computing and theInternet. Individuals and business users alike are provided with aseamlessly interoperable and Web-enabled interface for applications andcomputing devices, making computing activities increasingly Web browseror network-oriented.

For example, MICROSOFT®'s .NET™ platform, available from MicrosoftCorporation, includes servers, building-block services, such asWeb-based data storage, and downloadable device software. Whileexemplary embodiments herein are described in connection with softwareresiding on a computing device, one or more portions of an embodiment ofthe invention may also be implemented via an operating system,application programming interface (API) or a “middle man” object betweenany of a coprocessor, a display device and a requesting object, suchthat operation may be performed by, supported in or accessed via all of.NET™'s languages and services, and in other distributed computingframeworks as well.

As mentioned above, while exemplary embodiments of the invention havebeen described in connection with various computing devices and networkarchitectures, the underlying concepts may be applied to any computingdevice or system in which it is desirable to implement efficientvirtualization of memory in a virtual machine environment through ancache of address translations from multiple address spaces. Thus, themethods and systems described in connection with embodiments of thepresent invention may be applied to a variety of applications anddevices. While exemplary programming languages, names and examples arechosen herein as representative of various choices, these languages,names and examples are not intended to be limiting. One of ordinaryskill in the art will appreciate that there are numerous ways ofproviding object code that achieves the same, similar or equivalentsystems and methods achieved by embodiments of the invention.

The various techniques described herein may be implemented in connectionwith hardware or software or, where appropriate, with a combination ofboth. Thus, the methods and apparatus of the invention, or certainaspects or portions thereof, may take the form of program code (i.e.,instructions) embodied in tangible media, such as floppy diskettes,CD-ROMs, hard drives, or any other machine-readable storage medium,wherein, when the program code is loaded into and executed by a machine,such as a computer, the machine becomes an apparatus for practicing theinvention. In the case of program code execution on programmablecomputers, the computing device will generally include a processor, astorage medium readable by the processor (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. One or more programs that may utilizethe signal processing services of an embodiment of the presentinvention, e.g., through the use of a data processing API or the like,are preferably implemented in a high-level procedural or object-orientedprogramming language to communicate with a computer. However, theprogram(s) can be implemented in assembly or machine language, ifdesired. In any case, the language may be a compiled or interpretedlanguage, and combined with hardware implementations.

While aspects of the present invention have been described in connectionwith the preferred embodiments of the various figures, it is to beunderstood that other similar embodiments may be used or modificationsand additions may be made to the described embodiment for performing thesame function of the present invention without deviating therefrom.Furthermore, it should be emphasized that a variety of computerplatforms, including handheld device operating systems and otherapplication-specific operating systems are contemplated, especially asthe number of wireless networked devices continues to proliferate.Therefore, the claimed invention should not be limited to any singleembodiment, but rather should be construed in breadth and scope inaccordance with the appended claims.

1. A method of caching translations of virtual addresses (VA) to systemphysical addresses (SPA) from multiple address spaces in a virtualmachine, the method comprising: write-protecting a guest page tablecontaining at least one address translation for a first guest addressspace before caching the at least one address translation in a shadowpage table, wherein a subsequent write to the guest page table isintercepted; intercepting a write to the guest page table; creating aguest-writable mapping to the guest page table after intercepting atleast one write to the guest page table; wherein a subsequent write tothe guest page table is not intercepted and wherein the guest-writablemapping is a writable translation in a shadow page table from a virtualaddress to the system physical address of the guest page table; markinga cached address translation associated with the guest page tableaffected by the intercepted guest-writable mapping, wherein the markingindicates a stale cached address translation; purging the marked addresstranslations upon one of a switch to a second guest address space and aflush of translations, wherein non-stale address translations remaincached and stale address translations are removed; building a set oftracked guest pages dynamically, wherein the tracked guest pages containtracked guest page tables and are identified as guest page tables arewalked in resolving address translations; tracking every interceptedguest-writable mapping to a guest page in the set of tracked guestpages; removing all of the tracked guest-writable mappings to a trackedguest page, wherein a subsequent write to the tracked guest page isintercepted; counting a number of times that untracked guest page tablesare shadowed, wherein the untracked guest page tables haveguest-writable mappings that may not be tracked; deferringwrite-protecting a guest page containing an untracked guest page tablebeing shadowed until the number of times that untracked guest pagetables are shadowed exceeds a threshold, wherein the guest page isassumed to have guest-writable mappings; adding the guest pagecontaining an untracked guest page table being shadowed to a list ofguest pages that are candidates for being tracked; and purging allcached address translations when the number of times that untrackedguest page tables have been shadowed exceeds a threshold, wherein allguest pages are write-protected as a result of removing allguest-writable mappings and wherein the untracked guest pages on thecandidate list become tracked guest pages as a result ofwrite-protecting the untracked guest pages.
 2. A system for cachingtranslations of virtual addresses (VA) to system physical addresses(SPA) from multiple address spaces in a virtual machine, the systemcomprising; at least one guest program running on a host computer, theguest program and host computer comprising the virtual machine; at leastone data storage resource containing: at least one guest page containinga guest page table with at least one address translation; and at leastone host page containing a shadow page table with at least one addresstranslation; and a memory; and at least one host processor that accessesthe memory, the memory having instructions which when executed, performa method comprising: write-protecting a guest page table containing atleast one address translation for a first guest address space beforecaching the at least one address translation in a shadow page table,wherein a subsequent write to the guest page table is intercepted;intercepting a write to the guest page table; creating a guest-writablemapping to the guest page table after intercepting at least one write tothe guest page table; wherein a subsequent write to the guest page tableis not intercepted and wherein the guest-writable mapping is a writabletranslation in a shadow page table from a virtual address to the systemphysical address of the guest page table; marking a cached addresstranslation associated with the guest page table affected by theintercepted guest-writable mapping, wherein the marking indicates astale cached address translation; purging the marked addresstranslations upon one of a switch to a second guest address space and aflush of translations, wherein non-stale address translations remaincached and stale address translations are removed; building a set oftracked guest pages dynamically, wherein the tracked guest pages containtracked guest page tables and are identified as guest page tables arewalked in resolving address translations; tracking every interceptedguest-writable mapping to a guest page in the set of tracked guestpages; and removing all of the tracked guest-writable mappings to atracked guest page, wherein a subsequent write to the tracked guest pageis intercepted; and a software counter which counts a number of timesthat untracked guest page tables are shadowed, and wherein the hostprocessor executes software to: defer write protecting a guest pagecontaining an untracked guest page table being shadowed until the numberof times that untracked guest page tables are shadowed exceeds athreshold, wherein the guest page table is assumed to haveguest-writable mappings; add the guest page containing an untrackedguest page table being shadowed to a list of guest pages that arecandidates for being tracked; and purge all cached address translationswhen the number of times that untracked guest page tables have beenshadowed exceeds a threshold, wherein all guest pages arewrite-protected as a result of removing all guest-writable mappings andwherein the untracked guest pages on the candidate list become trackedguest pages as a result of write-protecting the untracked guest pages.3. A computer-readable storage medium, comprising instructions whichexecute a method of caching translations of virtual addresses (VA) tosystem physical addresses (SPA) from multiple address spaces in avirtual machine, the method comprising: write-protecting a guest pagetable containing at least one address translation for a first guestaddress space before caching the at least one address translation in ashadow page table, wherein a subsequent write to the guest page table isintercepted; intercepting a write to the guest page table; creating aguest-writable mapping to the guest page table after intercepting atleast one write to the guest page table; wherein a subsequent write tothe guest page table is not intercepted and wherein the guest-writablemapping is a writable translation in a shadow page table from a virtualaddress to the system physical address of the guest page table; markinga cached address translation associated with the guest page tableaffected by the intercepted guest-writable mapping, wherein the markingindicates a stale cached address translation; and purging the markedaddress translations upon one of a switch to a second guest addressspace and a flush of translations, wherein non-stale addresstranslations remain cached and stale address translations are removed;building a set of tracked guest pages dynamically, wherein the trackedguest pages contain tracked guest page tables and are identified asguest page tables are walked in resolving address translations; trackingevery intercepted guest-writable mapping to a guest page in the set oftracked guest pages; removing all of the tracked guest-writable mappingsto a tracked guest page, wherein a subsequent write to the tracked guestpage is intercepted; counting a number of times that untracked guestpage tables are shadowed, wherein the untracked guest page tables haveguest-writable mappings that may not be tracked; deferringwrite-protecting a guest page containing an untracked guest page tablebeing shadowed until the number of times that untracked guest pagetables are shadowed exceeds a threshold, wherein the guest page isassumed to have guest-writable mappings; adding the guest pagecontaining an untracked guest page table being shadowed to a list ofguest pages that are candidates for being tracked; and purging allcached address translations when the number of times that untrackedguest page tables have been shadowed exceeds a threshold, wherein allguest pages are write-protected as a result of removing allguest-writable mappings and wherein the untracked guest pages on thecandidate list become tracked guest pages as a result ofwrite-protecting the untracked guest pages.